GENERATIVE ADVERSARIAL NETWORKS (GANs) FOR SIMULATING SPECIMEN IMAGES

ABSTRACT

Methods and systems for generating a simulated image for a specimen are provided. One system includes one or more computer subsystems and one or more components executed by the one or more computer subsystems. The one or more components include a generative adversarial network (GAN), e.g., a conditional GAN (cGAN), trained with a training set that includes portions of design data for one or more specimens designated as training inputs and corresponding images of the one or more specimens designated as training outputs. The one or more computer subsystems are configured for generating a simulated image for a specimen by inputting a portion of design data for the specimen into the GAN.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention generally relates to methods and systems forgenerating a simulated image of a specimen using a generativeadversarial network (GAN).

2. Description of the Related Art

The following description and examples are not admitted to be prior artby virtue of their inclusion in this section.

Fabricating semiconductor devices such as logic and memory devicestypically includes processing a substrate such as a semiconductor waferusing a large number of semiconductor fabrication processes to formvarious features and multiple levels of the semiconductor devices. Forexample, lithography is a semiconductor fabrication process thatinvolves transferring a pattern from a reticle to a resist arranged on asemiconductor wafer. Additional examples of semiconductor fabricationprocesses include, but are not limited to, chemical-mechanical polishing(CMP), etch, deposition, and ion implantation. Multiple semiconductordevices may be fabricated in an arrangement on a single semiconductorwafer and then separated into individual semiconductor devices.

Inspection processes are used at various steps during a semiconductormanufacturing process to detect defects on wafers to drive higher yieldin the manufacturing process and thus higher profits. Inspection hasalways been an important part of fabricating semiconductor devices.However, as the dimensions of semiconductor devices decrease, inspectionbecomes even more important to the successful manufacture of acceptablesemiconductor devices because smaller defects can cause the devices tofail.

One significant hurdle in setting up most inspection processes isidentifying a sufficient number of defects of interest (DOIs) that canthen be used to set various parameters of the inspection processes. SuchDOIs are typically required for setting both hardware parameters such asoptical or other imaging parameters and software type parameters such asdefect classification settings, nuisance filter parameters, and thelike. If not enough DOI examples can be found on the setup specimen orspecimens, the resulting inspection recipe may be sub-optimal fordetecting and identifying those DOIs on other specimens.

While nuisance examples may also be required for setting up suchhardware and software parameters, nuisance examples tend to be readilyand overwhelmingly available. For example, when setting up a newinspection process, a hot scan may be performed on a setup specimen inwhich the threshold for defect detection is set at, near, or even in thenoise floor of the inspection system output. Therefore, such a scan mayproduce many more nuisance examples than needed, and those nuisances canmake identifying DOI examples particularly difficult because they needto be separated from all of the detected events, most of which arenuisances.

Another difficulty in setting up inspection processes is that the setupspecimen(s) may not contain any examples of one or more DOI types, whichcan result in an inspection process that is unable to detect such DOItype(s). Such a difficulty can also arise in maintaining performance ofan already setup inspection process. For example, if the DOI types onthe inspected specimens change for some reason, the already setupinspection process may not be able to detect the new DOI types. Such aninspection process may even become unusable, requiring an entirely newinspection process setup with a new setup specimen.

The lack of DOI examples can be particularly problematic for usingsophisticated techniques such as deep learning (DL) to perform nuisancereduction on an optical or other tool. As a result, inspection recipesmay be limited to manual tuning or random forest-based decision trees.However, even for random forest-type decision trees, a certain number ofDOI examples are required for training which are sometimes notavailable. Therefore, one current disadvantage to setting up inspectionand other recipes is that sophisticated techniques cannot be used fornuisance reduction in other inspection or quality control type processeswhen not enough example DOIs are available.

Accordingly, it would be advantageous to develop systems and methods forgenerating a simulated image of a specimen that do not have one or moreof the disadvantages described above.

SUMMARY OF THE INVENTION

The following description of various embodiments is not to be construedin any way as limiting the subject matter of the appended claims.

One embodiment relates to a system configured to generate a simulatedimage of a specimen. The system includes one or more computer subsystemsand one or more components executed by the one or more computersubsystems. The one or more components include a generative adversarialnetwork (GAN) trained with a training set that includes portions ofdesign data for one or more specimens designated as training inputs andcorresponding images of the one or more specimens designated as trainingoutputs. The one or more computer subsystems are configured forgenerating a simulated image for a specimen by inputting a portion ofdesign data for the specimen into the GAN. The system may be furtherconfigured as described herein.

Another embodiment relates to a computer-implemented method forgenerating a simulated image of a specimen. The method includesgenerating a simulated image for a specimen by inputting a portion ofdesign data for the specimen into a GAN. The inputting is performed byone or more computer subsystems. One or more components are executed bythe one or more computer subsystems. The one or more components includethe GAN. The GAN is trained with a training set that includes portionsof design data for one or more specimens designated as training inputsand corresponding images of the one or more specimens designated astraining outputs.

Each of the steps of the method may be further performed as describedfurther herein. The method may include any other step(s) of any othermethod(s) described herein. The method may be performed by any of thesystems described herein.

Another embodiment relates to a non-transitory computer-readable mediumstoring program instructions executable on one or more computer systemsfor performing a computer-implemented method for generating a simulatedimage for a specimen. The computer-implemented method includes the stepsof the method described above. The computer-readable medium may befurther configured as described herein. The steps of thecomputer-implemented method may be performed as described furtherherein. In addition, the computer-implemented method for which theprogram instructions are executable may include any other step(s) of anyother method(s) described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Further advantages of the present invention will become apparent tothose skilled in the art with the benefit of the following detaileddescription of the preferred embodiments and upon reference to theaccompanying drawings in which:

FIGS. 1 and 1 a are schematic diagrams illustrating side views ofembodiments of a system configured as described herein;

FIGS. 2-4 are flow charts illustrating steps that may be performed bythe embodiments described herein;

FIG. 5 is a schematic diagram illustrating one example of a generatorthat may be included in an embodiment of a generative adversarialnetwork (GAN);

FIG. 6 is a schematic diagram illustrating one example of adiscriminator that may be included in an embodiment of a GAN; and

FIG. 7 is a block diagram illustrating one embodiment of anon-transitory computer-readable medium storing program instructions forcausing computer system(s) to perform a computer-implemented methoddescribed herein.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. The drawingsmay not be to scale. It should be understood, however, that the drawingsand detailed description thereto are not intended to limit the inventionto the particular form disclosed, but on the contrary, the intention isto cover all modifications, equivalents and alternatives falling withinthe spirit and scope of the present invention as defined by the appendedclaims.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The terms “design,” “design data,” and “design information” as usedinterchangeably herein generally refer to the physical design (layout)of an IC or other semiconductor device and data derived from thephysical design through complex simulation or simple geometric andBoolean operations. The design may include any other design data ordesign data proxies described in commonly owned U.S. Pat. No. 7,570,796issued on Aug. 4, 2009 to Zafar et al. and U.S. Pat. No. 7,676,077issued on Mar. 9, 2010 to Kulkarni et al., both of which areincorporated by reference as if fully set forth herein. In addition, thedesign data can be standard cell library data, integrated layout data,design data for one or more layers, derivatives of the design data, andfull or partial chip design data. Furthermore, the “design,” “designdata,” and “design information” described herein refers to informationand data that is generated by semiconductor device designers in a designprocess and is therefore available for use in the embodiments describedherein well in advance of printing of the design on any physicalspecimens such as reticles and wafers.

“Nuisances” (which is sometimes used interchangeably with “nuisancedefects” or “nuisance events”) as that term is used herein is generallydefined as defects that a user does not care about and/or events thatare detected on a specimen but are not really actual defects on thespecimen. Nuisances that are not actually defects may be detected asevents due to non-defect noise sources on a specimen (e.g., grain inmetal lines on the specimen, signals from underlaying layers ormaterials on the specimen, line edge roughness (LER), relatively smallcritical dimension (CD) variation in patterned features, thicknessvariations, etc.) and/or due to marginalities in the inspection systemitself or its configuration used for inspection.

The term “defects of interest (DOIs)” as used herein is defined asdefects that are detected on a specimen and are really actual defects onthe specimen. Therefore, the DOIs are of interest to a user becauseusers generally care about how many and what kind of actual defects areon specimens being inspected. In some contexts, the term “DOI” is usedto refer to a subset of all of the actual defects on the specimen, whichincludes only the actual defects that a user cares about. For example,there may be multiple types of DOIs on any given specimen, and one ormore of them may be of greater interest to a user than one or more othertypes. In the context of the embodiments described herein, however, theterm “DOIs” is used to refer to any and all real defects on a specimen.

Turning now to the drawings, it is noted that the figures are not drawnto scale. In particular, the scale of some of the elements of thefigures is greatly exaggerated to emphasize characteristics of theelements. It is also noted that the figures are not drawn to the samescale. Elements shown in more than one figure that may be similarlyconfigured have been indicated using the same reference numerals. Unlessotherwise noted herein, any of the elements described and shown mayinclude any suitable commercially available elements.

In general, the embodiments described herein include a generativeadversarial network (GAN) or a type of GAN such as a conditional GAN(cGAN) for improved optical defect inspection performance and otherapplications. As described further herein, some embodiments create a newway to perform defect augmentation using a GAN or cGAN. Such embodimentstherefore enable the use of learning techniques that are based on and/orrequire significant numbers of defect examples. The embodimentsdescribed herein may also or alternatively be configured and used forgenerating reference images for any test image for applications such asdie-to-database type inspection.

In some embodiments, the specimen is a wafer. The wafer may include anywafer known in the semiconductor arts. Although some embodiments may bedescribed herein with respect to a wafer or wafers, the embodiments arenot limited in the specimens for which they can be used. For example,the embodiments described herein may be used for specimens such asreticles, flat panels, personal computer (PC) boards, and othersemiconductor specimens.

One embodiment relates to a system configured to generate a simulatedimage of a specimen. One embodiment of such a system is shown in FIG. 1.The system includes one or more computer subsystems 102 and one or morecomponents 104 executed by the one or more computer subsystems. In someembodiments, the system includes an inspection system coupled to the oneor more computer subsystems. For example, in FIG. 1, the system includesinspection system 100 coupled to computer subsystems(s) 102. In theembodiments shown in FIG. 1, the inspection system is configured as alight-based inspection system. However, in other embodiments describedherein, the inspection system is configured as an electron beam orcharged particle beam inspection system.

In general, the inspection systems described herein include at least anenergy source, a detector, and a scanning subsystem. The energy sourceis configured to generate energy that is directed to a specimen by theinspection system. The detector is configured to detect energy from thespecimen and to generate output responsive to the detected energy. Thescanning subsystem is configured to change a position on the specimen towhich the energy is directed and from which the energy is detected.

In a light-based inspection system, the energy directed to the specimenincludes light, and the energy detected from the specimen includeslight. In the embodiment of the system shown in FIG. 1, the inspectionsystem includes an illumination subsystem configured to direct light tospecimen 14. The illumination subsystem includes at least one lightsource, e.g., as shown in FIG. 1, light source 16. The illuminationsubsystem may be configured to direct the light to the specimen at oneor more angles of incidence, which may include one or more obliqueangles and/or one or more normal angles. For example, as shown in FIG.1, light from light source 16 is directed through optical element 18 andthen lens 20 to specimen 14 at an oblique angle of incidence. Theoblique angle of incidence may include any suitable oblique angle ofincidence, which may vary depending on, for instance, characteristics ofthe specimen and the defects to be detected on the specimen.

The illumination subsystem may be configured to direct the light to thespecimen at different angles of incidence at different times. Forexample, the inspection system may be configured to alter one or morecharacteristics of one or more elements of the illumination subsystemsuch that the light can be directed to the specimen at an angle ofincidence that is different than that shown in FIG. 1. In one suchexample, the inspection system may be configured to move light source16, optical element 18, and lens 20 such that the light is directed tothe specimen at a different oblique angle of incidence or a normal (ornear normal) angle of incidence.

In some instances, the inspection system may be configured to directlight to the specimen at more than one angle of incidence at the sametime. For example, the illumination subsystem may include more than oneillumination channel, one of the illumination channels may include lightsource 16, optical element 18, and lens 20 as shown in FIG. 1 andanother of the illumination channels (not shown) may include similarelements, which may be configured differently or the same, or mayinclude at least a light source and possibly one or more othercomponents such as those described further herein. If such light isdirected to the specimen at the same time as the other light, one ormore characteristics (e.g., wavelength, polarization, etc.) of the lightdirected to the specimen at different angles of incidence may bedifferent such that light resulting from illumination of the specimen atthe different angles of incidence can be discriminated from each otherat the detector(s).

In another instance, the illumination subsystem may include only onelight source (e.g., source 16 shown in FIG. 1) and light from the lightsource may be separated into different optical paths (e.g., based onwavelength, polarization, etc.) by one or more optical elements (notshown) of the illumination subsystem. Light in each of the differentoptical paths may then be directed to the specimen. Multipleillumination channels may be configured to direct light to the specimenat the same time or at different times (e.g., when differentillumination channels are used to sequentially illuminate the specimen).In another instance, the same illumination channel may be configured todirect light to the specimen with different characteristics at differenttimes. For example in some instances, optical element 18 may beconfigured as a spectral filter and the properties of the spectralfilter can be changed in a variety of different ways (e.g., by swappingout one spectral filter with another) such that different wavelengths oflight can be directed to the specimen at different times. Theillumination subsystem may have any other suitable configuration knownin the art for directing the light having different or the samecharacteristics to the specimen at different or the same angles ofincidence sequentially or simultaneously.

Light source 16 may include a broadband plasma (BBP) light source. Inthis manner, the light generated by the light source and directed to thespecimen may include broadband light. However, the light source mayinclude any other suitable light source such as a laser. The laser mayinclude any suitable laser known in the art and may be configured togenerate light at any suitable wavelength(s) known in the art. The lasermay be configured to generate light that is monochromatic ornearly-monochromatic. In this manner, the laser may be a narrowbandlaser. The light source may also include a polychromatic light sourcethat generates light at multiple discrete wavelengths or wavebands.

Light from optical element 18 may be focused onto specimen 14 by lens20. Although lens 20 is shown in FIG. 1 as a single refractive opticalelement, in practice, lens 20 may include a number of refractive and/orreflective optical elements that in combination focus the light from theoptical element to the specimen. The illumination subsystem shown inFIG. 1 and described herein may include any other suitable opticalelements (not shown). Examples of such optical elements include, but arenot limited to, polarizing component(s), spectral filter(s), spatialfilter(s), reflective optical element(s), apodizer(s), beam splitter(s),aperture(s), and the like, which may include any such suitable opticalelements known in the art. In addition, the system may be configured toalter one or more of the elements of the illumination subsystem based onthe type of illumination to be used for inspection.

The inspection system also includes a scanning subsystem configured tochange the position on the specimen to which the light is directed andfrom which the light is detected and possibly to cause the light to bescanned over the specimen. For example, the inspection system mayinclude stage 22 on which specimen 14 is disposed during inspection. Thescanning subsystem may include any suitable mechanical and/or roboticassembly (that includes stage 22) that can be configured to move thespecimen such that the light can be directed to and detected fromdifferent positions on the specimen. In addition, or alternatively, theinspection system may be configured such that one or more opticalelements of the inspection system perform some scanning of the lightover the specimen such that the light can be directed to and detectedfrom different positions on the specimen. In instances in which thelight is scanned over the specimen, the light may be scanned over thespecimen in any suitable fashion such as in a serpentine-like path or ina spiral path.

The inspection system further includes one or more detection channels.At least one of the detection channel(s) includes a detector configuredto detect light from the specimen due to illumination of the specimen bythe system and to generate output responsive to the detected light. Forexample, the inspection system shown in FIG. 1 includes two detectionchannels, one formed by collector 24, element 26, and detector 28 andanother formed by collector 30, element 32, and detector 34. As shown inFIG. 1, the two detection channels are configured to collect and detectlight at different angles of collection. In some instances, bothdetection channels are configured to detect scattered light, and thedetection channels are configured to detect light that is scattered atdifferent angles from the specimen. However, one or more of thedetection channels may be configured to detect another type of lightfrom the specimen (e.g., reflected light).

As further shown in FIG. 1, both detection channels are shown positionedin the plane of the paper and the illumination subsystem is also shownpositioned in the plane of the paper. Therefore, in this embodiment,both detection channels are positioned in (e.g., centered in) the planeof incidence. However, one or more of the detection channels may bepositioned out of the plane of incidence. For example, the detectionchannel formed by collector 30, element 32, and detector 34 may beconfigured to collect and detect light that is scattered out of theplane of incidence. Therefore, such a detection channel may be commonlyreferred to as a “side” channel, and such a side channel may be centeredin a plane that is substantially perpendicular to the plane ofincidence.

Although FIG. 1 shows an embodiment of the inspection system thatincludes two detection channels, the inspection system may include adifferent number of detection channels (e.g., only one detection channelor two or more detection channels). In one such instance, the detectionchannel formed by collector 30, element 32, and detector 34 may form oneside channel as described above, and the inspection system may includean additional detection channel (not shown) formed as another sidechannel that is positioned on the opposite side of the plane ofincidence. Therefore, the inspection system may include the detectionchannel that includes collector 24, element 26, and detector 28 and thatis centered in the plane of incidence and configured to collect anddetect light at scattering angle(s) that are at or close to normal tothe specimen surface. This detection channel may therefore be commonlyreferred to as a “top” channel, and the inspection system may alsoinclude two or more side channels configured as described above. Assuch, the inspection system may include at least three channels (i.e.,one top channel and two side channels), and each of the at least threechannels has its own collector, each of which is configured to collectlight at different scattering angles than each of the other collectors.

As described further above, each of the detection channels included inthe inspection system may be configured to detect scattered light.Therefore, the inspection system shown in FIG. 1 may be configured fordark field (DF) inspection of specimens. However, the inspection systemmay also or alternatively include detection channel(s) that areconfigured for bright field (BF) inspection of specimens. In otherwords, the inspection system may include at least one detection channelthat is configured to detect light specularly reflected from thespecimen. Therefore, the inspection systems described herein may beconfigured for only DF, only BF, or both DF and BF inspection. Althougheach of the collectors are shown in FIG. 1 as single refractive opticalelements, it is to be understood that each of the collectors may includeone or more refractive optical element(s) and/or one or more reflectiveoptical element(s).

The one or more detection channels may include any suitable detectorsknown in the art such as photo-multiplier tubes (PMTs), charge coupleddevices (CCDs), and time delay integration (TDI) cameras. The detectorsmay also include non-imaging detectors or imaging detectors. If thedetectors are non-imaging detectors, each of the detectors may beconfigured to detect certain characteristics of the light such asintensity but may not be configured to detect such characteristics as afunction of position within the imaging plane. As such, the output thatis generated by each of the detectors included in each of the detectionchannels may be signals or data, but not image signals or image data. Insuch instances, a computer subsystem such as computer subsystem 36 ofthe inspection system may be configured to generate images of thespecimen from the non-imaging output of the detectors. However, in otherinstances, the detectors may be configured as imaging detectors that areconfigured to generate imaging signals or image data. Therefore, theinspection system may be configured to generate images in a number ofways.

It is noted that FIG. 1 is provided herein to generally illustrate aconfiguration of an inspection system that may be included in the systemembodiments described herein. Obviously, the inspection systemconfiguration described herein may be altered to optimize theperformance of the inspection system as is normally performed whendesigning a commercial inspection system. In addition, the systemsdescribed herein may be implemented using an existing inspection system(e.g., by adding functionality described herein to an existinginspection system) such as the 29xx/39xx series of tools that arecommercially available from KLA Corp., Milpitas, Calif. For some suchsystems, the methods described herein may be provided as optionalfunctionality of the inspection system (e.g., in addition to otherfunctionality of the inspection system).

Alternatively, the inspection system described herein may be designed“from scratch” to provide a completely new inspection system.

Computer subsystem 36 may be coupled to the detectors of the inspectionsystem in any suitable manner (e.g., via one or more transmission media,which may include “wired” and/or “wireless” transmission media) suchthat the computer subsystem can receive the output generated by thedetectors. Computer subsystem 36 may be configured to perform a numberof functions using the output of the detectors. For instance, thecomputer subsystem may be configured to detect events on the specimenusing the output of the detectors. Detecting the events on the specimenmay be performed by applying some defect detection algorithm and/ormethod to the output generated by the detectors, which may include anysuitable algorithm and/or method known in the art. For example, thecomputer subsystem may compare the output of the detectors to athreshold. Any output having values above the threshold may beidentified as an event (e.g., a potential defect) while any outputhaving values below the threshold may not be identified as an event.

The computer subsystem of the inspection system may be furtherconfigured as described herein. For example, computer subsystem 36 maybe part of the one or more computer subsystems described herein or maybe configured as the one or more computer subsystems described herein.In particular, computer subsystem 36 may be configured to perform thesteps described herein. As such, the steps described herein may beperformed “on-tool,” by a computer system or subsystem that is part ofan inspection system.

The computer subsystem of the inspection system (as well as othercomputer subsystems described herein) may also be referred to herein ascomputer system(s). Each of the computer subsystem(s) or system(s)described herein may take various forms, including a personal computersystem, image computer, mainframe computer system, workstation, networkappliance, Internet appliance, or other device. In general, the term“computer system” may be broadly defined to encompass any device havingone or more processors, which executes instructions from a memorymedium. The computer subsystem(s) or system(s) may also include anysuitable processor known in the art such as a parallel processor. Inaddition, the computer subsystem(s) or system(s) may include a computerplatform with high speed processing and software, either as a standaloneor a networked tool.

If the system includes more than one computer subsystem, then thedifferent computer subsystems may be coupled to each other such thatimages, data, information, instructions, etc. can be sent between thecomputer subsystems. For example, computer subsystem 36 may be coupledto computer subsystems(s) 102 as shown by the dashed line in FIG. 1 byany suitable transmission media, which may include any suitable wiredand/or wireless transmission media known in the art. Two or more of suchcomputer subsystems may also be effectively coupled by a sharedcomputer-readable storage medium (not shown).

Although the inspection system is described above as being an optical orlight-based inspection system, in another embodiment, the inspectionsystem is configured as an electron beam inspection system. In anelectron beam inspection system, the energy directed to the specimenincludes electrons, and the energy detected from the specimen includeselectrons. In one such embodiment shown in FIG. 1a , the inspectionsystem includes electron column 122, and the system includes computersubsystem 124 coupled to the inspection system. Computer subsystem 124may be configured as described above. In addition, such an inspectionsystem may be coupled to another one or more computer subsystems in thesame manner described above and shown in FIG. 1.

As also shown in FIG. 1a , the electron column includes electron beamsource 126 configured to generate electrons that are focused to specimen128 by one or more elements 130. The electron beam source may include,for example, a cathode source or emitter tip, and one or more elements130 may include, for example, a gun lens, an anode, a beam limitingaperture, a gate valve, a beam current selection aperture, an objectivelens, and a scanning subsystem, all of which may include any suchsuitable elements known in the art.

Electrons returned from the specimen (e.g., secondary electrons) may befocused by one or more elements 132 to detector 134. One or moreelements 132 may include, for example, a scanning subsystem, which maybe the same scanning subsystem included in element(s) 130.

The electron column may include any other suitable elements known in theart. In addition, the electron column may be further configured asdescribed in U.S. Pat. No. 8,664,594 issued Apr. 4, 2014 to Jiang etal., U.S. Pat. No. 8,692,204 issued Apr. 8, 2014 to Kojima et al., U.S.Pat. No. 8,698,093 issued Apr. 15, 2014 to Gubbens et al., and U.S. Pat.No. 8,716,662 issued May 6, 2014 to MacDonald et al., which areincorporated by reference as if fully set forth herein.

Although the electron column is shown in FIG. 1a as being configuredsuch that the electrons are directed to the specimen at an oblique angleof incidence and are scattered from the specimen at another obliqueangle, the electron beam may be directed to and scattered from thespecimen at any suitable angles. In addition, the electron beaminspection system may be configured to use multiple modes to generateoutput for the specimen as described further herein (e.g., withdifferent illumination angles, collection angles, etc.). The multiplemodes of the electron beam inspection system may be different in anyoutput generation parameters of the inspection system.

Computer subsystem 124 may be coupled to detector 134 as describedabove. The detector may detect electrons returned from the surface ofthe specimen thereby forming electron beam images of (or other outputfor) the specimen. The electron beam images may include any suitableelectron beam images. Computer subsystem 124 may be configured to detectevents on the specimen using output generated by detector 134, which maybe performed as described above or in any other suitable manner.Computer subsystem 124 may be configured to perform any additionalstep(s) described herein. A system that includes the inspection systemshown in FIG. 1a may be further configured as described herein.

It is noted that FIG. 1a is provided herein to generally illustrate aconfiguration of an electron beam inspection system that may be includedin the embodiments described herein. As with the optical inspectionsystem described above, the electron beam inspection systemconfiguration described herein may be altered to optimize theperformance of the inspection system as is normally performed whendesigning a commercial inspection system. In addition, the systemsdescribed herein may be implemented using an existing inspection system(e.g., by adding functionality described herein to an existinginspection system) such as the tools that are commercially availablefrom KLA. For some such systems, the methods described herein may beprovided as optional functionality of the system (e.g., in addition toother functionality of the system). Alternatively, the system describedherein may be designed “from scratch” to provide a completely newsystem.

Although the inspection system is described above as being a light orelectron beam inspection system, the inspection system may be an ionbeam inspection system. Such an inspection system may be configured asshown in FIG. 1a except that the electron beam source may be replacedwith any suitable ion beam source known in the art. In addition, theinspection system may include any other suitable ion beam imaging systemsuch as those included in commercially available focused ion beam (FIB)systems, helium ion microscopy (HIM) systems, and secondary ion massspectroscopy (SIMS) systems.

As further noted above, the inspection system may be configured to havemultiple modes. In general, a “mode” is defined by the values ofparameters of the inspection system used to generate output for thespecimen. Therefore, modes that are different may be different in thevalues for at least one of the optical or electron beam parameters ofthe inspection system (other than position on the specimen at which theoutput is generated). For example, for a light-based inspection system,different modes may use different wavelengths of light. The modes may bedifferent in the wavelengths of light directed to the specimen asdescribed further herein (e.g., by using different light sources,different spectral filters, etc.) for different modes. In anotherembodiment, different modes may use different illumination channels. Forexample, as noted above, the inspection system may include more than oneillumination channel. As such, different illumination channels may beused for different modes.

The multiple modes may also be different in illumination and/orcollection/detection. For example, as described further above, theinspection system may include multiple detectors. Therefore, one of thedetectors may be used for one mode and another of the detectors may beused for another mode. Furthermore, the modes may be different from eachother in more than one way described herein (e.g., different modes mayhave one or more different illumination parameters and one or moredifferent detection parameters). The inspection system may be configuredto scan the specimen with the different modes in the same scan ordifferent scans, e.g., depending on the capability of using multiplemodes to scan the specimen at the same time.

The systems described herein may be configured as another type ofsemiconductor-related process/quality control type system such as adefect review system and a metrology system. For example, theembodiments of the systems described herein and shown in FIGS. 1 and 1 amay be modified in one or more parameters to provide different imagingcapability depending on the application for which they will be used. Inone embodiment, the electron beam inspection system configurationdescribed herein may be modified to be configured as an electron beamdefect review system. For example, the system shown in FIG. 1a may beconfigured to have a higher resolution if it is to be used for defectreview or metrology rather than for inspection. In other words, theembodiments of the system shown in FIGS. 1 and 1 a describe some generaland various configurations for a system that can be tailored in a numberof manners that will be obvious to one skilled in the art to producesystems having different imaging capabilities that are more or lesssuitable for different applications.

As noted above, the inspection systems may be configured for directingenergy (e.g., light, electrons) to and/or scanning energy over aphysical version of the specimen thereby generating actual images forthe physical version of the specimen. In this manner, the inspectionsystems may be configured as “actual” imaging systems, rather than“virtual” systems. A storage medium (not shown) and computersubsystem(s) 102 shown in FIG. 1 may be configured as a “virtual”system. Systems and methods configured as “virtual” inspection systemsare described in commonly assigned U.S. Pat. No. 8,126,255 issued onFeb. 28, 2012 to Bhaskar et al. and U.S. Pat. No. 9,222,895 issued onDec. 29, 2015 to Duffy et al., both of which are incorporated byreference as if fully set forth herein. The embodiments described hereinmay be further configured as described in these patents.

The one or more components executed by the one or more computersubsystems include generative adversarial network (GAN) 106 shown inFIG. 1. The GAN is trained with a training set that includes portions ofdesign data for one or more specimens designated as training inputs andcorresponding images of the one or more specimens designated as trainingoutputs. The GAN may or may not be trained by the one or more computersubsystems and/or one of the component(s) executed by the computersubsystem(s). For example, another method or system may train the GAN,which then may be stored for use as the components executed by thecomputer subsystem(s).

Some steps that may be performed by the one or more computer subsystemsto train the GAN are shown in FIG. 2. Although FIG. 2 shows steps thatmay be performed in a method for using a GAN for data augmentation, someof the steps shown in FIG. 2 may be performed for other applications aswell. For example, steps 200, 202, and 204 may be performed to train aGAN regardless of which of the applications described herein the GANwill be used for.

As shown in step 200, the computer subsystem(s) may align optical imageto design using pixel-to-design alignment (PDA). If the GAN is beingtrained for a different kind of image, the optical image(s) may bereplaced in the steps shown in FIG. 2 with another type of (e.g.,electron beam) image. PDA may be performed in any suitable manner knownin the art, some examples of which are described in the above-referencedpatent to Kulkarni et al. PDA may be performed for any one or morealignment target(s) on the specimen to thereby determine image-to-designoffsets, which can be then be used by the computer subsystem(s) toidentify which portion of a design for the specimen corresponds to animage generated for the specimen and/or an event detected on thespecimen.

As shown in step 202, the computer subsystem(s) may run a hot scan toobtain pairs of optical image and design clip. A “design clip” as thatterm is used herein is generally defined as a relatively small portionof an entire design for a specimen. The term “design clip” is usedinterchangeably herein with the phrase “portion of design data.” The hotscan may be performed in any suitable manner known in the art. Ingeneral, a “hot” scan involves scanning a specimen and detecting eventson the specimen using a “hot” threshold, e.g., a threshold that is at,near, or even in the noise floor of the output generated by thescanning. For any events that are detected in the hot scan, the computersubsystem(s) may create pairs of optical images and their correspondingdesign clips. The pairs may be created for only some or all of theevents detected in the hot scan. In general, the pairs may be createdfor fewer than all of the events because hot scans generally will detectmany more events than are needed for any one application. The computersubsystem(s) may sample or select which events are used to create imageand design clip pairs in any suitable manner. The image and design clippairs may also be created for locations at which no events weredetected. In this manner, the training set may include image and designclip pairs for “good” locations on the specimen(s). In some suchembodiments, the hot scan may not detect events on the specimen at all,but may simply collect images that can then be paired with correspondingdesign clips. Whether for events or not, the corresponding images anddesign clips can be identified and paired using the image-to-designoffsets described above.

The image and design clip pairs generated by step 202 may then be usedas a training set, with the design clips designated as the traininginputs and corresponding images designed as the training outputs, instep 204 in which the GAN (i.e., the generator-discriminator network(NW)) is trained using design and optical images. The training of theGAN may be performed as described further herein. In some embodiments,the computer subsystem(s) may be configured to perform steps 200, 202,and 204. As another alternative, different computer subsystem(s) orsystem(s) may be configured to perform steps 200, 202, and 204. Forexample, a first computer subsystem may be configured to generate thetraining set by performing steps 200 and 202, a second computersubsystem may be configured to train the GAN in step 204, and the firstand second computer subsystems may or may not be included in thecomputer subsystem(s) of the embodiments described herein.

The training may include inputting the training inputs into the GAN andaltering one or more parameters of the GAN until the output produced bythe GAN matches (or substantially matches) the training outputs.Training may include altering any one or more trainable parameters ofthe GAN. For example, the one or more parameters of the GAN that aretrained by the embodiments described herein may include one or moreweights for any layer of the GAN that has trainable weights. In one suchexample, the weights may include weights for convolution layers but notpooling layers.

In one embodiment, the GAN is configured as a conditional GAN (cGAN). Ingeneral, GANs consist of two adversarial models, a generative model, G,capturing the data distribution, and a discriminative model, D,estimating the probability that a given sample comes from the trainingdata rather than G. G and D could be a multi-layer perceptron, i.e. anon-linear mapping function. The generator builds a mapping functionfrom a prior noise distribution p_(z)(Z) to the data space G(z; θ_(g))in order to learn a generator distribution p_(g) over the data x where Gis a differentiable function represented by a multilayer perceptron withparameters θ_(g). The generator is trained to produce images that cannotbe distinguished from real images. The adversarially traineddiscriminator is trained to detect fakes which are created by thegenerator. Both the generator and the discriminator are trained as goodas possible so that the generator produces extremely good “faked”images.

When extending GANs to a conditional model (cGAN), the discriminator andthe generator are both conditioned on some extra information, y, byfeeding y into the discriminator and the generator as additional inputlayer.

GANs are inspired by game theory where a generator G and a critic, i.e.,Discriminator D, are competing with each other to make each otherstronger. The goal is to make the generated data distribution p_(g) assimilar as possible to the real sample distribution p_(r) evaluated bythe Jensen-Shannon divergence:

${D_{JS}\left( {p_{g}\text{||}_{Pr}} \right)} = {{\frac{1}{2}{D_{KL}\left( {p_{g}\text{||}\frac{p_{g} + p_{r}}{2}} \right)}} + {\frac{1}{2}{D_{KL}\left( {p_{r}\text{||}\frac{p_{g} + p_{r}}{2}} \right)}}}$

where D_(KL) is the Kullback-Leibler divergence defined as follows:

${D_{KL}\left( {p_{g}\text{||}\frac{p_{g} + p_{r}}{2}} \right)} = {{\int_{x_{1}}^{x_{2}}{{p_{g}(x)}\log\frac{2{p_{g}(x)}}{{p_{g}(x)} + {p_{r}(x)}}{dx}\mspace{14mu}{and}\mspace{14mu}{D_{KL}\left( {p_{r}\text{||}\frac{p_{g} + p_{r}}{2}} \right)}}} = {\int_{x_{1}}^{x_{2}}{{p_{r}(x)}\log\frac{2{p_{r}(x)}}{{p_{g}(x)} + {p_{r}(x)}}{dx}}}}$

The generator outputs synthetic samples given a random noise variableinput z. Over time, the generator gets trained to capture the real datadistribution by having the discriminator reject images which itconsiders as being bad fakes.

For conditional GANs, the Generator and Discriminator are conditioned onsome extra information, y, which will be the design clip in theembodiments described herein. The following minimax game with theobjective L(D,G) is describing the setting:

${\min\limits_{G}{\max\limits_{D}{L\left( {D,\ G} \right)}}} = {{{\mathbb{E}}_{x \sim {p_{r}{(x)}}}\left( {\log\;{D\left( {x\text{|}y} \right)}} \right)} + {{\mathbb{E}}_{z \sim {p_{z}{(z)}}}\left( {\log\left( {1 - {{D\left( {G\left( {z\text{|}y} \right)} \right)}{\quad{= {{{\mathbb{E}}_{x \sim {p_{r}{(x)}}}\left( {\log\;{D\left( {x\text{|}y} \right)}} \right)} + {{\mathbb{E}}_{x \sim {p_{g}{(x)}}}\left( {\log\left( {1 - {D\left( {x\text{|}y} \right)}} \right)} \right)}}}}}} \right.} \right.}}$

This means that G is trained to increase chances of D producing a highprobability for a faked example, thus minimizing

_(x˜p) _(g) _((x))(log(1−D(x|y)). We also need to make sure that thediscriminator decisions over real data are accurate by maximizing theencoding

_(x˜p) _(r) _((x))(log D(x|y)) and given a fake sample G(z), thediscriminator is expected to output a probability, D(G(z)), close tozero by maximizing

_(z˜p) _(z) _((z))(log(1−D(G(z|y))).

FIG. 3 shows cGAN training performed by providing the generator with adesign image. In particular, as shown in FIG. 3, extra information y 300(a design image or conditional image) is input to generator network 302along with latent space vector z 301. As a result, generator network 302outputs generated patch image 304. Original patch image x 306, in thiscase, is an actual optical image of a specimen at the location on thespecimen at which the portion of the design shown in design image 300 isformed. In this manner, extra information y 300 such as a design imagemay be a training input in a training set, and original patch image x306 may be its corresponding image of a specimen designated as atraining output in the training set. Generated patch image 304 and extrainformation y 300 are combined as a first input to discriminator network310. Original patch image x 306 and extra information y 308 (a designimage) are combined as a second input to discriminator network 310.Extra information y 300 and 308 are merely different instances of thesame portion of the same design in this case. The discriminator networkmay generate output P(True) 312, which is the probability that thegenerated patch image is a true or good “faked” image. P(True) may beinput to loss function 314.

Each of the design images, original patch images, and generated patchimages shown in the figures is not meant to illustrate any particularspecimen(s) or characteristics thereof for which the embodimentsdescribed herein may be used. In a similar manner, each of the originalpatch images and generated patch images shown in the figures is notmeant to illustrate any particular actual or simulated images that maybe generated for specimen(s). Instead, the design images, original patchimages, and generated patch images shown in the figures are merelyintended to promote understanding of the embodiments described herein.The images actually input and output from the generator will varydepending on the specimen and its characteristics, which are related toits design, and the configuration of the imaging system that generatesthe actual images for the specimen(s), which are used to train the GANthereby affecting the simulated images that are generated by the GAN.

Loss function 314 may be any suitable loss function known in the artsuch as that disclosed by Isola et al., in “Image-to-Image Translationwith Conditional Adversarial Networks,” The IEEE Conference on ComputerVision and Pattern Recognition (CVPR), 2017, pp. 1125-1134, which isincorporated by reference as if fully set forth herein. The embodimentsdescribed herein may be further configured as described in thisreference.

An example of a generator that may be included in embodiments of a GANconfigured as described herein is shown in FIG. 5. In general, the GANmay include an encoder-decoder network that is progressively downsampling until a bottle neck layer and then the process is reversed. Asshown in FIG. 5, the generator may include encoder 500 and decoder 502.Each of the blocks 506, 508, and 510 shown in encoder 500 represent anexample of final output layer size after repeated convolution, batchnormalization, and rectified linear unit (ReLU) activation and applyingmax pooling in the end of each section. Although encoder 500 is shown inFIG. 5 as including 3 blocks, the encoder may include any suitablenumber of blocks, which may be determined in any suitable manner knownin the art. In addition, each of the blocks, convolution layer(s), batchnormalization layer(s), ReLU layer(s), and pooling layer(s) may have anysuitable configuration known in the art. Input 504, which is in theembodiments described herein a portion of design data, may be input intoblock 506, whose output may be input to block 508, and so on. Theencoder may generate feature layer 512.

The decoder may also include multiple blocks that perform differentfunctions on feature layer 512 input to the decoder. Each of blocks 514,516, and 518 in the decoder represent an example of final output layersize after repeating upsampling (transposed convolution) and ReLUactivation. Although decoder 502 is shown in FIG. 5 as including 3blocks, the decoder may include any suitable number of blocks, which maybe determined in any suitable manner known in the art. Each of theblocks and upsampling and ReLU layer(s) included in the decoder may haveany suitable configuration known in the art. Feature layer 512 generatedby the encoder may be input to block 514, whose output may be input toblock 516, and so on. Output 520 of the decoder may be any of thesimulated images described herein.

In some instances, the GAN may include skip connections 522 betweencorresponding blocks in the encoder and decoder, e.g., between blocks506 and 518, between blocks 508 and 516, and between blocks 510 and 514.Connections can be skipped to transfer low-level information that hasbeen learned between the blocks. The skip connections may have anysuitable configuration determined in any suitable manner known in theart. The numbers below the input and output in FIG. 5 indicate the sizeof the input and output, respectively. The numbers below the blocks inthe encoder indicate the size of the outputs of the blocks, and thenumbers below the blocks in the decoder indicate the size of the inputsto each of the blocks.

An example of a discriminator that may be included in embodiments of aGAN configured as described herein is shown in FIG. 6. Input 600 to thediscriminator may include two images (not shown in FIG. 6), onegenerated by the generator of the GAN and the original image. Thediscriminator may include a number of layers including layers 602, 604,606, 608, and 610, each of which may include some combination ofconvolution, ReLU, and max pooling layers. The convolution, ReLU, andmax pooling layers may have any suitable configuration known in the art.Output 612 of the discriminator may be P(x), the probability of how wellthe simulated image matches the original image or the probability thatthe simulated image is a good “faked” image or a bad “faked” image. Thenumbers below the input in FIG. 6 indicate the size of the input. Thenumbers below the layers in the discriminator indicate the size of theoutputs of the layers. Although the discriminator is shown in FIG. 6with a particular number of layers, the discriminator included in theGANs of the embodiments described herein may have any suitable number oflayers determined in any suitable manner known in the art.

Additional description of the general architecture and configuration ofGANs and cGANs can be found in “Generative Adversarial Nets” Goodfellowet al., arXiv:1406.2661, Jun. 10, 2014, 9 pages and “ConditionalGenerative Adversarial Nets,” Mirza et al., arXiv:1411.1784, Nov. 6,2014, 7 pages, which are incorporated by reference as if fully set forthherein. The embodiments described herein may be further configured asdescribed in these references.

The one or more computer subsystems are configured for generating asimulated image for a specimen by inputting a portion of design data forthe specimen into the GAN. The computer subsystem(s) may be configuredto input the portion of the design data for the specimen into the GAN inany suitable manner known in the art. Although some embodiments aredescribed herein (only for the sake of clarity and simplicity) asgenerating “a simulated image” for a specimen, the GANs described hereinmay be used to generate any number of simulated images limited only bythe portions of design data input to the GANs. In addition, althoughsome embodiments are described herein (again only for the sake ofclarity and simplicity) as generating a simulated image for “aspecimen,” the embodiments described herein are not limited togenerating simulated image(s) for only one specimen. In someembodiments, generating simulated image(s) for only one specimen may beall that is required for some applications such as when the simulatedimage(s) are reference image(s) that can be used for inspection ofmultiple specimens or when the simulated image(s) are augmented defectimages that are used to train another deep learning (DL) or machinelearning (ML) model or network.

In one embodiment, the simulated image is a simulated optical image. Inthis manner, the simulated image illustrates how the specimen appears inone or more actual images generated by an optical imaging system such asan optical inspection system. As such, the simulated image(s) mayrepresent (e.g., correspond, simulate, or approximate) images that maybe generated of the specimen by an optical imaging system. Suchsimulated optical images may be used in a variety of ways describedfurther herein for inspection applications. One novel feature of theembodiments described herein is therefore that they provide a method forimproved optical defect inspection using a GAN or cGAN for theartificial creation of optical patch images.

In another embodiment, the specimen for which the simulated image isgenerated is not included in the one or more specimens. For example, theone or more specimens that are used to generate the training data thatis then used to train a GAN may be different from the specimen for whichthe GAN is used to generate a simulated image. In this manner, thetraining data may be generated using one or more training specimens, andthe simulated image may be generated for a “runtime” specimen. Thetraining specimen(s) and the runtime specimen may be the same type ofspecimens, e.g., they may have the same design and may have beenprocessed using the same fabrication process step(s), although asdescribed further herein that is not necessarily always the case. Inaddition, the trained GAN may be used for generating simulated imagesfor more than one runtime specimen.

In a further embodiment, a design for the specimen for which thesimulated image is generated is different than one or more designs forthe one or more specimens. For example, as described above, the specimenfor which the simulated image is generated may not be the same as (orincluded in) the one or more specimens used to train the GAN. Ingeneral, the one or more specimens used to train the GAN and thespecimen for which the image is simulated may have the same design andmay have been processed in the same processes (and therefore be of thesame “layer”). Training the GAN in such a manner will ensure that thesimulated image(s) most closely resemble the actual images.

But in some cases, a first specimen may have similar enoughcharacteristics (e.g., patterned features, materials, etc.) to a secondspecimen, that a GAN trained on the first specimen can be used togenerate simulate image(s) for the second specimen even if the first andsecond specimens do not have the same designs. In such cases, thesimulated image(s) that are generated by the trained GAN should be forthe same optical mode as that for which it was trained. In other words,as described further herein, a GAN that is trained for generatingsimulated image(s) produced by one imaging mode may not necessarily besuitable for generating simulated image(s) produced by another imagingmode. Therefore, if two specimens that have at least some similaritiesin at least a portion of their designs are or will be imaged in the samemanner, one of the specimens may be used to train the GAN and thetrained GAN may be used to generate simulated image(s) for another ofthe specimens.

In this manner, a trained GAN may be repurposed for generating simulatedimage(s) for specimens it was not necessarily trained for. In one suchexample, if two different specimens with two different designs have atleast some patterned features in common in a portion of the design(e.g., similar memory array areas) formed of similar materials andhaving the same or similar dimensions, a GAN trained for one of thespecimens may be capable of producing simulated image(s) of syntheticdefects in that portion of the design for another of the specimens. Evenif a GAN trained for one specimen is not capable of producing simulatedimage(s) for another specimen having a different design, if there aresome similarities among the specimens, the trained GAN may be used asstarting configuration that is re-trained for another specimen to createa different GAN. Such re-training may be performed as described herein.

In one embodiment, the simulated image includes an augmented defectimage of a synthetic defect. In this manner, the simulated imageillustrates how a synthetic defect appears in one or more actual imagesgenerated by a tool such as an inspection system. As such, the simulatedimage(s) may represent (e.g., correspond, simulate, or approximate)image(s) that may be generated of the synthetic defect(s) by theinspection system or other tool. A “synthetic” defect as that term isused herein is generally defined as a defect that is artificiallycreated in a design for a specimen in one of several manners describedherein. Therefore, a “synthetic” defect is not a defect that has beendetected on a specimen although its characteristics may be determinedbased on such a detected defect. Instead, such a “synthetic” defect iscreated in a design for a specimen through the intentional manipulationof an otherwise defect free portion of the design. When such amanipulated portion of the design is input to a trained GAN describedherein, the resulting simulated image is referred to as an “augmented”defect image since the simulated specimen image has been augmented witha defect that is not actually present on the specimen.

FIG. 4 illustrates one embodiment of steps that may be performed forartificial image generation using a GAN. In particular, as shown in FIG.4, the one or more computer subsystems may input design image 400(conditional image) with added defect 402 into trained generator network404. The added or synthetic defect may be created in the design dataportion shown in design image 400 as described further herein, which maybe performed by the one or more computer subsystems or another system ormethod. The trained generator network may output generated patch image406 showing defect 408. In this manner, the synthetic defect is addeddefect 402 in the portion of design data shown in design image 400,which is input to the GAN by the one or more computer subsystems tothereby generate simulated image 406.

As shown in FIG. 4, therefore, once the GAN is trained, the GAN can beused to generate simulated optical and other images. In this embodiment,the trained generator network is used to create a patch, real-lookingimage of an artificially introduced defect in a design clip. Thegenerated patch images can then be used for data augmentation, e.g., toaugment other non-simulated patch images and information to trainanother ML network, which may be performed as described further herein.

In one such embodiment, a number of actual defects detectable on the oneor more specimens and the specimen by an inspection system isinsufficient for training a ML model. For example, the embodimentsdescribed herein can be used to generate simulated image(s) that areaugmented defect image(s) of synthetic defect(s) that can then be usedto augment training data, which is particularly advantageous when thenumber of DOI events is limited which is very often the case. Anotheradvantage of the embodiments described herein is therefore that theyenable the use of ML techniques such as DL, random forest, etc. evenwhen the original number of DOI examples is substantially low. Theembodiments described herein also advantageously enable the use of MLtechniques when there are no DOI examples available, e.g., as describedfurther herein, when there are no optical images of a DOI at all,simulated images may be generated by the embodiments described hereinbased on knowledge of how the DOI is supposed to look in design spacewithin design clip(s), and those simulated images may then be used totrain a ML model.

In another such embodiment, the one or more computer subsystems areconfigured for generating the portion of the design data for thespecimen input to the GAN by modifying original design data for thespecimen with the synthetic defect. For example, as shown in step 206 ofFIG. 2, the one or more computer subsystems may select DOI candidatesfrom scanning electron microscope (SEM) reviewed hot scan and locatethem in design. In one such example, if a hot scan is performed as shownin FIG. 2 for GAN training, events detected by the hot scan may besampled and then sent for defect review, e.g., by a SEM or anothersuitable defect review tool. The defect review is performed to determinewhich of the sampled events are actual defects and which are nuisancesand to determine a defect type or classification of the actual defects.In this manner, defect review may determine which of the detected eventsare DOIs and identify the type or classification of the DOI (when it ispossible that different types of DOIs may be present on a specimen).Step 206 is an optional step since, in some cases, a user will know whattype(s) of DOI(s) they are interested in. For example, a chipmanufacturer will usually know what defects they are looking for.

As shown in step 208 of FIG. 2, the one or more computer subsystems maythen introduce the same defect type into design clips from the hot scan.If the specimen for which the augmented defect image(s) are beinggenerated has the same design as the one or more specimens used forgenerating the training set, then the design clips used for steps 200,202, and 204 may also be used for step 208. If the specimen for whichthe augmented defect image(s) are being generated has a different designthan the one or more specimens used for generating the training set,then the design clips used for steps 200, 202, and 204 may be differentfrom those used for step 208.

In either case, however, the DOI(s) that are selected in step 206 may beused in step 208 to create design clip(s) that are modified to includethe DOI(s). For example, if a bridge type DOI is selected in step 206,the original design data for the specimen may be modified by thecomputer subsystem(s) to include bridge type structure(s) at location(s)in the design that are different than the location at which the bridgetype DOI was detected. The location(s) at which the DOI is created in adesign may be selected based on the types of patterned features locatedat or near the detected DOI, e.g., so that the modified design clip andthe original design clip at the location of a detected DOI contain thesame or at least similar patterned structures (similar in shape,dimensions, spacing relative to each other, orientation, etc.). In oneexample, if a bridge type DOI is detected between two patterned featureshaving a particular orientation and spacing relative to each other, abridge type patterned feature may be created at any one or more otherlocations in the design containing the same two patterned featureshaving the same relative orientation and spacing. Such locations may beidentified in any suitable manner, e.g., by pattern searching theoriginal design based on the image of the detected DOI or the original,defect-free design clip at the location of the detected DOI.

Modifying the original design data for the specimen may be performed asdescribed herein using an electron design automation (EDA) tool. The EDAtool may include any suitable commercially available EDA tool. Inaddition, modifying the original design data as described herein can beautomated with a programmable/graphical EDA editor, which may includeany suitable EDA software, hardware, system, or method. In some suchembodiments, one or more of the computer subsystems described herein(e.g., computer subsystem(s) 102) may be configured as an EDA tool ormay be a computer subsystem included in an EDA tool.

If the original design data (e.g., computer-aided design (CAD)) for thespecimen is available, it is straightforward to inject “legal,”synthetic defect examples. For example, DOIs such as opens, shorts,“mouse bites,” protrusions, etc. can be rendered (drawn) with varioussizes, which could be automated based on descriptions of the DOIs. Usingan EDA tool, these rendered DOIs can be located in “legal” places in thegeometry instead of in random places. In one example, a short is a metalconnection between two copper lines. For such a DOI, one could simplyadd a small short line at strategic pinch points in the design. When auser such as a chip manufacturer knows what defects they are lookingfor, they could also manually draw them in the design file.

Modifying the original design data for the specimen with the syntheticdefect as described herein may also be performed using an inceptionmodule configured for altering the design to create the synthetic defectin the design. For example, a neural network may be trained by a defecthallucination system such as those suggested by GoogLeNet inception fornatural scene images. A traditional neural network that is pre-trainedon defects can then play these backwards to create new defect types onother geometry structures. Examples of systems and methods forperforming GoogLeNet inception can be found in “Going Deeper withConvolutions,” Szegedy et al., 2015 IEEE Conference on Computer Visionand Pattern Recognition (CVPR), June 2015, 9 pages, which isincorporated by reference as if fully set forth herein. The embodimentsdescribed herein may be further configured as described in thisreference.

As shown in step 210, the one or more computer subsystems may thenprocess the design clips generated by step 208 through the previouslytrained GAN. The trained GAN may then output artificially generatedimages, as shown in step 212. Among other things, the images output bythe GAN in step 212 may be used as input data for training otherlearning algorithms, as shown in step 214, which may be performed asdescribed further herein.

In some such embodiments, the one or more computer subsystems areconfigured for determining one or more characteristics of the syntheticdefect based on one or more defects detected on the one or morespecimens. For example, as described above, in some instances, eventsmay be detected on the one or more specimens used for generating thetraining set that is used to train the GAN, and DOI(s) may be identifiedand selected from among those events. Information for the DOI(s) maythen be used to determine one or more characteristics of the syntheticdefects. Such information may include any information that is or can bedetermined for the DOI(s) such as location, size, shape, orientation,texture or roughness, patterned feature(s) on which the DOI(s) arelocated, patterned feature(s) in which the DOI(s) are located, patternedfeature(s) located near the DOI(s), etc. Such information may bedetermined or generated by the inspection tool that detected the events,a review tool that re-detected the events and identified one or more ofthem as DOI(s), a metrology tool that measures one or morecharacteristics of the identified DOI(s), or some combination thereof.For example, as described above, pattern searching may be performedbased on a defect image or its corresponding design clip to find othersimilar instances of a pattern in a design, and then the design forthose other instances may be modified as described herein to create theDOI(s) at those other instances. Other DOI characteristic(s) such asthose described above may also be used to create the modified design(e.g., in the case of a bridge type DOI, the dimensions, orientation,roughness, etc. of the bridge type structure). Therefore, the originaldesign for the specimen may be modified based on one or morecharacteristics of one or more DOI(s) detected on the specimen(s) usedto generate the training set.

In another such embodiment, the one or more computer subsystems areconfigured for determining one or more characteristics of the syntheticdefect without information for one or more actual defects detected onthe one or more specimens or the specimen. For example, even if thereare no DOI examples at all, the one or more computer subsystems canmodify a design clip for the specimen by introducing a defect in it andinput the modified design clip into the GAN to create the correspondingpatch image that can then be used to train a neural network forapplications such as nuisance filtering. One novel feature of theembodiments described herein therefore is that they provide a method forimproved optical defect inspection using a GAN or cGAN to modify adesign file by introducing a defect in it and generating a correspondingoptical patch image even when there are no defect examples at all.

When there are no defect examples at all, the one or morecharacteristics of the synthetic defect may be determined in variouspossible ways. For example, as described above, in many instances inwhich the embodiments described herein may be implemented, a user mayhave knowledge about which types of DOI(s) that are of interest to them.Such knowledge may be based on prior knowledge about the types of DOI(s)typically seen on specimens produced in the same or similar process(es)as the specimen for which the simulated image is generated. Such priorknowledge may be acquired experimentally, heuristically, theoretically,etc. For example, even when designs of specimens are substantiallydifferent, certain patterned structures that may be in each of thedesigns may be prone to certain type(s) of DOI(s). In one particularexample, alternating line/space patterns in which the lines and spacesare relatively small in width may be prone to bridging type defectsformed between the lines. Therefore, when a user sees such a patterngroup in a new design, they may surmise that bridging type defects maybe present in such patterns and therefore may create such defects in thedesign for the specimen. The design clips created artificially for suchdefects may then be input to the GAN by the computer subsystem(s)described herein to thereby generate simulated images for the defects.

In a further such embodiment, the synthetic defect is a first type ofDOI, the one or more computer subsystems are configured for generatingan additional portion of the design data for the specimen by modifyingthe original design data for the specimen with an additional syntheticdefect, the additional synthetic defect is a second type of DOIdifferent than the first type, and the one or more computer subsystemsare configured for generating an additional simulated image for thespecimen by inputting the additional portion of the design data into theGAN. For example, the embodiments described herein can advantageously beused to easily generate simulated images for different DOI typeexamples. In other words, the GANs described herein can generatesimulated images for different types of DOIs in the same design for thesame specimen.

Many of the specimens described herein may have multiple types of DOIspresent on them, and the user may be interested in more than one (oreven all) of those multiple types of DOIs. Different DOIs may havesubstantially different characteristics. For example, bridge typedefects can have substantially different characteristics than pinholetype defects, but both types of defects may be present on a specimen andpossibly of interest to a user. Different types of DOIs may be locatedin different portions of the design or in the same portion of thedesign. Therefore, a design clip that is modified as described herein toinclude a synthetic defect may include a single synthetic defect ormultiple synthetic defects. Regardless of the number of syntheticdefects in a modified design clip and/or the number of modified designclips, which may be for multiple instances of the same DOI type and/orfor different DOI types, a GAN trained as described herein will becapable of producing substantially high quality simulated images foreach of the DOI types and for each DOI instance.

In an additional such embodiment, the one or more computer subsystemsare configured for determining one or more characteristics of thesynthetic defect based on one or more defects detected on one or moreadditional specimens, the one or more additional specimens are formed inone or more process steps, a change in the one or more process stepsoccurs prior to the one or more process steps being used to form thespecimen, a ML model was trained to perform one or more functions forthe one or more additional specimens, and the one or more computersubsystems are configured to re-train the ML model using the simulatedimage. In this manner, one advantage of the embodiments described hereinis that they can be used to mitigate process variation by generatingimages with DOI examples from a first wafer and image examples from asecond wafer.

In one such example, suppose that a false positive filter, oftenreferred to as a nuisance event filter or NEF, has been setup on wafer1. In some instances, the process may change and thus defect attributesmay change for a later wafer, referred to here simply as wafer 2. Thatchange in defect attributes may mean that the filter setup on wafer 1 isuseless for wafer 2 but also since wafer 2 was not the setup wafer,there may not be any defect examples for wafer 2 for retraining thefilter. The implication here is that it will not be possible to trainthe NEF classifier, which could be, for example, a random forest model.However, the system embodiments described herein can solve this problemby collecting some image data on wafer 2, e.g., using an inspectionsystem such as that described herein. In this case, no new DOIinformation is needed. The one or more computer subsystem(s) may thenuse these images from wafer 2 to train a new GAN or retrain thepreviously trained GAN, which may be performed as described furtherherein and shown in FIGS. 2 and 3, using design and corresponding patchimages. The one or more computer subsystem(s) may then introduce asynthetic defect in a design clip to generate a modified design clipthat can be input to the newly trained or retrained GAN to therebygenerate a corresponding simulated defect image. Such simulated defectimage(s) may be used as described further herein to train another DL orML model or network.

In one embodiment, the corresponding images of the one or more specimensare generated by a first mode of an imaging system, the one or morecomponents include an additional GAN trained with an additional trainingset that includes the portions of the design data for the one or morespecimens designated as additional training inputs and correspondingadditional images of the one or more specimens designated as additionaltraining outputs, the corresponding additional images are generated by asecond mode of the imaging system different than the first mode, and theone or more computer subsystems are configured for generating anadditional simulated image for the specimen by inputting the portion ofthe design data for the specimen into the additional GAN.

In such embodiments, therefore, different GANs may be trained to produceimages or other output for different modes. In particular, in mostcases, different modes of an inspection or other imaging tool willproduce images and/or output that are different from each other in oneof several possible ways, e.g., noise levels, contrast, resolution,image type (e.g., DF vs. BF, optical vs. electron beam, etc.), and thelike. Therefore, if a GAN is trained to produce output or images for onemode of a tool, chances are it will be unsuitably trained to produceoutput or simulated images for another mode of the tool. As such,multiple GANs may be separately and independently trained, one for eachmode of interest. The same pre-trained GAN network may however be usedfor each mode although that is not necessary. Each trained GAN may thenbe used to generate a mode specific data set. The input to eachdifferently trained GAN, however, may be the same design clips ordifferent portions of the same design since the design of the specimenwill not change from mode to mode.

If the GANs are trained or used to generate simulated defect images, thesimulated images generated by the trained GANs for different modes maythen be used to train one or more DL or ML models or networks. Forexample, if the embodiments described herein generate or augment atraining data set with the simulated image(s) generated by the GANs,that training data set can be used to train a neural network for singleand multiple optical (or other) modes, which may be performed asdescribed in U.S. Pat. No. 10,115,040 to Brauer issued Oct. 30, 2018 andU.S. Pat. No. 10,360,477 to Bhaskar et al. issued Jul. 23, 2019, whichare incorporated by reference as if fully set forth herein. Theembodiments described herein may be further configured as described inthese patents. Whether one or more DL or ML models or networks aretrained using the training set may depend on the intended use of the DLor ML model(s) or network(s) and what their inputs are. For example, oneDL or ML model or network may be configured to perform nuisancefiltering or defect classification using multiple mode images as inputs.Therefore, simulated images generated for the multiple modes may be usedto train the model or network. In another example, one DL or ML model ornetwork may be configured to perform nuisance filtering using imagesfrom a first mode as inputs, and a different DL or ML model or networkmay be configured to perform defect classification using images from asecond mode as inputs. In this manner, simulated images generated forthe first mode may be used to train one DL or ML model or network, andsimulated images generated for the second mode may be used to trainanother DL or ML model or network.

If the GANs are trained or used to generate simulated reference images,different simulated reference images may be generated and stored asdescribed further herein for different modes. For example, if defectdetection is to be performed in a die-to-database manner, each mode mayrequire its own reference image that is subtracted from test imagesgenerated in each mode. Therefore, when different GANs are trained asdescribed herein for different modes, the different GANs may be used togenerate different reference images, each of which may be used in thesame inspection process but for different modes.

In some embodiments, the one or more computer subsystems are configuredto train a ML model using the simulated image. In some such embodiments,the additional generated samples, e.g., simulated images, can be usedfor augmenting training data sets or by adding to other augmentationbased defect examples produced by augmentation techniques such as thosedescribed in U.S. Pat. No. 10,402,688 to Brauer et al. issued Sep. 3,2019, which is incorporated by reference as if fully set forth herein.The embodiments described herein may be further configured as describedin this patent. The simulated images may be added to a training setand/or used to create a new training set in any suitable manner known inthe art. The ML model that is trained using the simulated image(s)described herein may have any suitable configuration known in the art.Training the ML model using the simulated image(s) generated asdescribed herein may otherwise be performed as described herein or inany other suitable manner known in the art.

In one such embodiment, the ML model is configured for nuisancefiltering. One advantage of the embodiments described herein is that theapproach described herein or a derivation of it allows reduction ofnuisance which in turn allows semiconductor manufacturers to make morereliable process decisions and thus not waste money using the incorrectprocessing conditions. In another such embodiment, the ML model isconfigured for defect classification. In this manner, the ML model mayidentify different types of defects and assign them to different classesor groups. A ML model that is trained using the simulated imagesdescribed herein may have any suitable configuration known in the artwhether it is configured for nuisance filtering or defectclassification.

In another embodiment, the simulated image is a reference image, and thesystem includes an inspection system configured for detecting defects onthe specimen by subtracting the reference image from images of thespecimen generated by the inspection system. For example, if thesimulated image(s) are generated for use in die-to-database typeinspections, the one or more computer subsystems may input every givendesign clip or full frame design or even a full die design into thetrained GAN to thereby generate a reference image. This artificial imagemay then be used as a reference image for defect inspection. In thismanner, the embodiments described herein can be used to generate anartificial optical (or other) reference image to perform die-to-databaseinspections. One novel feature of the embodiments described herein istherefore that they provide a method for improved optical defectinspection using a GAN or cGAN for the artificial creation of opticalreference patch images which can be used for optical die-to-databaseinspections. Another novel feature of the embodiments described hereinis that they provide a method for improved optical defect inspectionusing a GAN or cGAN for an unsupervised DL method for optical referenceimage generation.

The computer subsystem(s) may be configured for storing a variety ofinformation, images, etc. generated by the embodiments described herein.For example, the computer subsystem(s) may be configured for storing thesimulated augmented defect image(s) generated by the GAN for use intraining of a ML or DL model or network such as those described herein.In one such example, the computer subsystem(s) may store the simulateddefect image(s) in a data structure or file containing the training setthat is being augmented by the simulated defect image(s). The simulateddefect image(s) may be stored in any suitable manner and in anycomputer-readable storage medium described herein.

In another example, the computer subsystem(s) may be configured forstoring the simulated reference image and/or a ML or DL model or networktrained using a training set augmented with one or more simulated imagesgenerated by the embodiments described herein for use in inspection ofthe specimen or another specimen of the same type. The computersubsystem(s) may be configured to store such image(s) and/or model ornetwork in a recipe or by generating a recipe for the inspection inwhich the reference image(s) and/or model or network will be used. A“recipe” as that term is used herein is defined as a set of instructionsthat can be used by a tool to perform a process on a specimen. In thismanner, generating a recipe may include generating information for how aprocess is to be performed, which can then be used to generate theinstructions for performing that process. The computer subsystem(s) mayalso store any information for the reference image(s) and/or model ornetwork that can be used to identify, access, and/or use the image(s)and/or model or network (e.g., such as a file name and where it isstored). The information for the model or network that is stored mayalso include the code, instructions, algorithms, etc. for the model ornetwork. The reference image(s), model or network, or informationtherefor may be stored in any suitable manner in any of thecomputer-readable storage media described herein.

The images, model or network, and information therefor may be storedwith any of the other results described herein and may be stored in anymanner known in the art. The storage medium may include any storagemedium described herein or any other suitable storage medium known inthe art. After the information has been stored, the information can beaccessed in the storage medium and used by any of the method or systemembodiments described herein, formatted for display to a user, used byanother software module, method, or system, etc. For example, theembodiments described herein may generate an inspection recipe asdescribed above. That inspection recipe may then be stored and used bythe system or method (or another system or method) to inspect thespecimen or other specimens to thereby generate information (e.g.,defect information) for the specimen or other specimens.

Results and information generated by performing the inspection on thespecimen or other specimens of the same type may be used in a variety ofmanners by the embodiments described herein and/or other systems andmethods. Such functions include, but are not limited to, altering aprocess such as a fabrication process or step that was or will beperformed on the inspected specimen or another specimen in a feedback orfeedforward manner. For example, the computer subsystem(s) describedherein may be configured to determine one or more changes to a processthat was performed on a specimen inspected as described herein and/or aprocess that will be performed on the specimen based on the detecteddefect(s). The changes to the process may include any suitable changesto one or more parameters of the process. The computer subsystem(s)described herein preferably determine those changes such that thedefects can be reduced or prevented on other specimens on which therevised process is performed, the defects can be corrected or eliminatedon the specimen in another process performed on the specimen, thedefects can be compensated for in another process performed on thespecimen, etc. The computer subsystem(s) described herein may determinesuch changes in any suitable manner known in the art.

Those changes can then be sent to a semiconductor fabrication system(not shown) or a storage medium (not shown) accessible to the computersubsystem(s) and the semiconductor fabrication system. The semiconductorfabrication system may or may not be part of the system embodimentsdescribed herein. For example, the computer subsystem(s) and/orinspection system described herein may be coupled to the tosemiconductor fabrication system, e.g., via one or more common elementssuch as a housing, a power supply, a specimen handling device ormechanism, etc. The semiconductor fabrication system may include anysemiconductor fabrication system known in the art such as a lithographytool, an etch tool, a chemical-mechanical polishing (CMP) tool, adeposition tool, and the like.

As described herein, therefore, the embodiments can be used to setup anew inspection process or recipe. The embodiments may also be used tomodify an existing inspection process or recipe, whether that is aninspection process or recipe that was used for the specimen or wascreated for one specimen and is being adapted for another specimen.

The embodiments described herein are not limited to inspection recipe orprocess creation or modification. For example, the embodiments describedherein can also be used to setup or modify a recipe or process formetrology, defect review, etc. in a similar manner. In particular, theGANs described herein can be trained depending on the process that isbeing setup or revised (e.g., to generate simulated outputs that mimicthe actual outputs that would be generated by the process). Then,depending on the process or recipe that is being setup or altered, thesimulated outputs may be used to setup a recipe for that process,whether that is storing a simulated reference image that is used in theprocess or to train a DL or ML model or network for use in the process.Such output processing methods may include, for example, defectre-detection methods used for re-detecting defects in output generatedby a defect review system.

In a similar manner, the embodiments described herein may be used toselect not just output processing parameters and methods but also outputacquisition parameters or modes, which with, for example, an inspectionsystem, a metrology system, or a defect review system detects light,electrons, ions, etc. from a specimen. Such output acquisition parameterselection may include training and using different GANs to generatesimulated images for different output acquisition parameters or modes,which may be performed as described further herein. The generatedsimulated images can then be compared and evaluated to select whichmodes or parameters are the best for any one process. The embodimentsdescribed herein can therefore be used not just for setting up ormodifying an inspection process but also for setting up or modifying anyquality control type process performed on the specimens described hereinand any parameters of such a process.

Each of the embodiments of each of the systems described above may becombined together into one single embodiment.

Another embodiment relates to a computer-implemented method forgenerating a simulated image of a specimen. The method includesgenerating a simulated image for a specimen by inputting a portion ofdesign data for the specimen into a GAN. The inputting is performed byone or more computer subsystems. One or more components are executed bythe one or more computer subsystems. The one or more components includethe GAN. The GAN is trained with a training set that includes portionsof design data for one or more specimens designated as training inputsand corresponding images of the one or more specimens designated astraining outputs.

Each of the steps of the method may be performed as described furtherherein. The method may also include any other step(s) that can beperformed by the system, computer subsystem(s), component(s), and/orGANs described herein. The computer subsystem(s) may be configuredaccording to any of the embodiments described herein, e.g., computersubsystem(s) 102. The one or more components and the GAN may also beconfigured according to any of the embodiments described herein. Themethod may be performed by any of the system embodiments describedherein.

An additional embodiment relates to a non-transitory computer-readablemedium storing program instructions executable on one or more computersystems for performing a computer-implemented method for generating asimulated image of a specimen. One such embodiment is shown in FIG. 7.In particular, as shown in FIG. 7, non-transitory computer-readablemedium 700 includes program instructions 702 executable on computersystem(s) 704. The computer-implemented method may include any step(s)of any method(s) described herein.

Program instructions 702 implementing methods such as those describedherein may be stored on computer-readable medium 700. Thecomputer-readable medium may be a storage medium such as a magnetic oroptical disk, a magnetic tape, or any other suitable non-transitorycomputer-readable medium known in the art.

The program instructions may be implemented in any of various ways,including procedure-based techniques, component-based techniques, and/orobject-oriented techniques, among others. For example, the programinstructions may be implemented using ActiveX controls, C++ objects,JavaBeans, Microsoft Foundation Classes (“MFC”), SSE (Streaming SIMDExtension) or other technologies or methodologies, as desired.

Computer system(s) 704 may be configured according to any of theembodiments described herein.

Further modifications and alternative embodiments of various aspects ofthe invention will be apparent to those skilled in the art in view ofthis description. For example, methods and systems for generating asimulated image for a specimen are provided. Accordingly, thisdescription is to be construed as illustrative only and is for thepurpose of teaching those skilled in the art the general manner ofcarrying out the invention. It is to be understood that the forms of theinvention shown and described herein are to be taken as the presentlypreferred embodiments. Elements and materials may be substituted forthose illustrated and described herein, parts and processes may bereversed, and certain features of the invention may be utilizedindependently, all as would be apparent to one skilled in the art afterhaving the benefit of this description of the invention. Changes may bemade in the elements described herein without departing from the spiritand scope of the invention as described in the following claims.

What is claimed is:
 1. A system configured to generate a simulated imageof a specimen, comprising: one or more computer subsystems; and one ormore components executed by the one or more computer subsystems, whereinthe one or more components comprise a generative adversarial networktrained with a training set comprising portions of design data for oneor more specimens designated as training inputs and corresponding imagesof the one or more specimens designated as training outputs; and whereinthe one or more computer subsystems are configured for generating asimulated image for a specimen by inputting a portion of design data forthe specimen into the generative adversarial network.
 2. The system ofclaim 1, wherein the generative adversarial network is configured as aconditional generative adversarial network.
 3. The system of claim 1,wherein the simulated image comprises an augmented defect image of asynthetic defect.
 4. The system of claim 3, wherein a number of actualdefects detectable on the one or more specimens and the specimen by aninspection system is insufficient for training a machine learning model.5. The system of claim 3, wherein the one or more computer subsystemsare further configured for generating the portion of the design data forthe specimen input to the generative adversarial network by modifyingoriginal design data for the specimen with the synthetic defect.
 6. Thesystem of claim 5, wherein the one or more computer subsystems arefurther configured for determining one or more characteristics of thesynthetic defect based on one or more defects detected on the one ormore specimens.
 7. The system of claim 5, wherein the one or morecomputer subsystems are further configured for determining one or morecharacteristics of the synthetic defect without information for one ormore actual defects detected on the one or more specimens or thespecimen.
 8. The system of claim 5, wherein the synthetic defect is afirst type of defect of interest, wherein the one or more computersubsystems are further configured for generating an additional portionof the design data for the specimen by modifying the original designdata for the specimen with an additional synthetic defect, wherein theadditional synthetic defect is a second type of defect of interestdifferent than the first type, and wherein the one or more computersubsystems are further configured for generating an additional simulatedimage for the specimen by inputting the additional portion of the designdata into the generative adversarial network.
 9. The system of claim 5,wherein the one or more computer subsystems are further configured fordetermining one or more characteristics of the synthetic defect based onone or more defects detected on one or more additional specimens,wherein the one or more additional specimens are formed in one or moreprocess steps, wherein a change in the one or more process steps occursprior to the one or more process steps being used to form the specimen,wherein a machine learning model was trained to perform one or morefunctions for the one or more additional specimens, and wherein the oneor more computer subsystems are further configured to re-train themachine learning model using the simulated image.
 10. The system ofclaim 1, wherein the corresponding images of the one or more specimensare generated by a first mode of an imaging system, wherein the one ormore components further comprise an additional generative adversarialnetwork trained with an additional training set comprising the portionsof the design data for the one or more specimens designated asadditional training inputs and corresponding additional images of theone or more specimens designated as additional training outputs, whereinthe corresponding additional images are generated by a second mode ofthe imaging system different than the first mode, and wherein the one ormore computer subsystems are further configured for generating anadditional simulated image for the specimen by inputting the portion ofthe design data for the specimen into the additional generativeadversarial network.
 11. The system of claim 1, wherein the specimen forwhich the simulated image is generated is not included in the one ormore specimens.
 12. The system of claim 1, wherein a design for thespecimen for which the simulated image is generated is different thanone or more designs for the one or more specimens.
 13. The system ofclaim 1, wherein the one or more computer subsystems are furtherconfigured to train a machine learning model using the simulated image.14. The system of claim 13, wherein the machine learning model isconfigured for nuisance filtering.
 15. The system of claim 13, whereinthe machine learning model is configured for defect classification. 16.The system of claim 1, wherein the simulated image is a reference image,and wherein the system further comprises an inspection system configuredfor detecting defects on the specimen by subtracting the reference imagefrom images of the specimen generated by the inspection system.
 17. Thesystem of claim 1, wherein the simulated image is a simulated opticalimage.
 18. The system of claim 1, wherein the specimen is a wafer.
 19. Anon-transitory computer-readable medium, storing program instructionsexecutable on one or more computer systems for performing acomputer-implemented method for generating a simulated image of aspecimen, wherein the computer-implemented method comprises: generatinga simulated image for a specimen by inputting a portion of design datafor the specimen into a generative adversarial network, wherein saidinputting is performed by the one or more computer systems; wherein oneor more components are executed by the one or more computer systems,wherein the one or more components comprise the generative adversarialnetwork; and wherein the generative adversarial network is trained witha training set comprising portions of design data for one or morespecimens designated as training inputs and corresponding images of theone or more specimens designated as training outputs.
 20. Acomputer-implemented method for generating a simulated image of aspecimen, comprising: generating a simulated image for a specimen byinputting a portion of design data for the specimen into a generativeadversarial network, wherein said inputting is performed by one or morecomputer subsystems; wherein one or more components are executed by theone or more computer subsystems, wherein the one or more componentscomprise the generative adversarial network; and wherein the generativeadversarial network is trained with a training set comprising portionsof design data for one or more specimens designated as training inputsand corresponding images of the one or more specimens designated astraining outputs.